LARGE MULTIMODAL MODELS

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation (2025.naacl-long)

Challenge:	Programming often involves translating detailed and complex specifications into code . current state-of-the-art models struggle to solve these problems, a new study shows .
Approach:	They propose a multi-modal coding dataset to evaluate algorithmic problem-solving skills in visually rich contexts.
Outcome:	The proposed model lacks powerful vision-code models due to the extreme demand for reasoning abilities.

Copied to clipboard

Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa

APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention (2026.acl-long)

Challenge:	Using culture-agnostic subsets, performance drops in many LMMs when evaluated in Japanese.
Approach:	They introduce a Japanese benchmark to evaluate large multimodal models on expert-level tasks based on the Japanese cultural context.
Outcome:	The proposed benchmark enables comparisons with other benchmarks in other languages based on cultural contexts.

Copied to clipboard

Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan Liu

Harnessing PDF Data for Improving Japanese Large Multimodal Models (2025.findings-acl)

Challenge:	Existing methods for long-video inference use compression or sparse attention . existing methods restrict LMMs from handling longer, more complex videos .
Approach:	They propose a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs.
Outcome:	The proposed framework delivers speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB without significant performance loss.

Copied to clipboard

Jeonghun Baek, Akiko Aizawa, Kiyoharu Aizawa

Can Large Multimodal Models Uncover Deep Semantics Behind Images? (2024.findings-acl)

Challenge:	Large Multimodal Models (LMMs) have demonstrated strong performance in English, but their effectiveness in Japanese remains limited due to the lack of high-quality training data.
Approach:	They propose a pipeline that leverages pretrained models to extract image-text pairs from PDFs . they use layout analysis, OCR, and vision-language pairing to enrich the training data .
Outcome:	The proposed pipeline extracts image-text pairs from Japanese PDFs, eliminating manual annotations.

Copied to clipboard

Yixin Yang, Zheng Li, Qingxiu Dong, Heming Xia, Zhifang Sui

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning (2025.findings-acl)

Challenge:	Existing studies on visual deep semantics focus primarily on superficial description of images, revealing a notable deficiency in the systematic investigation of the inherent deep semantic.
Approach:	They propose a benchmark to assess Large Multimodal Models’ (LMMs) capacities of visual deep semantics.
Outcome:	The proposed benchmark demonstrates a substantial gap between the deep semantic comprehension capabilities of existing LMMs and humans.

Copied to clipboard

Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, Hongsheng Li

Towards Low-Resource Harmful Meme Detection with LMM Agents (2024.emnlp-main)

Challenge:	Large Language Models (LMMs) struggle with simple tasks such as geometry, e.g., arithmetic, and reasoning.
Approach:	They propose to leverage code as supervision for cross-modal alignment . they propose to use FigCodifier and ImgCode-8.6M to synthesize novel mathematical figures .
Outcome:	The proposed model surpasses GPT-4o and Claude 3.5 Sonnet in the geometry problem-solving subset of MathVista, achieving improvements of 8.9% and 9.2%.

Copied to clipboard

Jianzhao Huang, Hongzhan Lin, Liu Ziyan, Ziyang Luo, Guang Chen, Jing Ma

The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning (2025.naacl-long)

Challenge:	Existing methods for harmful meme detection are limited due to the dynamic nature of memes . eliciting knowledge-revising behavior within the LMM agent is a key factor in achieving this goal .
Approach:	They propose an agency-driven framework for low-resource harmful meme detection . they use annotated memes to leverage label information as auxiliary signals for model .
Outcome:	The proposed framework achieves superior performance than state-of-the-art methods on the low-resource harmful meme detection task.

Copied to clipboard

Longju Bai, Angana Borah, Oana Ignat, Rada Mihalcea

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024.acl-long)

Challenge:	Large Multimodal Models exhibit impressive performance across multimodal tasks . effectiveness in cross-cultural contexts limited due to predominantly Western-centric nature of data and models . multi-agent models have shown significant capability in solving complex tasks despite limitations in crosscultural context .
Approach:	They propose to use a multi-agent framework to enhance cross-cultural image captioning using LMMs with distinct cultural personas to evaluate cultural information within image captions.
Outcome:	The proposed model outperforms single-agent models across different metrics and offers valuable insights for future research.

Copied to clipboard

Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, Maosong Sun

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation (2025.naacl-long)

Challenge:	Large Language Models (LLMs) and Large Multimodal Models have exceeded general human capabilities in various tasks.
Approach:	They present an Olympiad-level bilingual multimodal scientific benchmark featuring 8,476 problems from Olympiad level mathematics and physics competitions.
Outcome:	The best performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning.

Copied to clipboard

Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, Ming Zhang

M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks (2024.findings-emnlp)

Challenge:	Large Multimodal Models (LMMs) exhibit impressive cross-modal understanding and reasoning abilities, but many benchmarks suffer from systematic biases.
Approach:	They propose a benchmark to avoid Type-I errors by creating one perception question and one knowledge anchor question through a meticulous annotation process.
Outcome:	The proposed benchmark avoids Type-I errors while maintaining reliability of MCQ evaluations.

Copied to clipboard

Florian Schneider, Sunayana Sitaram

ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback (2024.emnlp-main)

Challenge:	Recent advances in Large Language Models and their multimodal counterparts have shown significant performance disparities across different languages and cultural contexts.
Approach:	They propose to evaluate LLMs on diverse vision-language tasks within a multilingual and multicultural context using M5 benchmark.
Outcome:	The proposed benchmarks highlight task-agnostic performance disparities between languages and cultural contexts.

Copied to clipboard

Ju-Seung Byun, Jiyun Chun, Jihyung Kil, Andrew Perrault

UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents (2026.acl-long)

Challenge:	Large Multimodal Models excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks.
Approach:	They propose an algorithm that alters REinforcement Learning and Supervised Fine-Tuning to refine large multimodal models with specific preferences.
Outcome:	The proposed algorithm achieves 70% win rate compared to baseline models judged by GPT-4o.

Copied to clipboard

Yifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Qian Zhang, Zhibo Yang, Junyang Lin, Yu Gu, Ge Yu, Maosong Sun

Information Extraction & Text Mining

Challenge:	Recent Large Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images.
Approach:	They propose a benchmark to evaluate the performance of Large Multimodal Models (LMMs) using a constrained-category KIE track and an open-categorical KIE Track.
Outcome:	Experiments on 15 state-of-the-art LMMs show performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios.

A Context-Aware Contrastive Learning Framework for Hateful Meme Detection and Segmentation (2025.findings-naacl)

Copied to clipboard

Xuanyu Su, Yansong Li, Diana Inkpen, Nathalie Japkowicz

PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)

Challenge:	Empirical experiments show HateSieve surpasses existing LMMs in performance with fewer trainable parameters .
Approach:	They propose a framework to enhance detection and segmentation of hateful elements in memes by creating a triplet dataset and an Image-Text Alignment module.
Outcome:	HateSieve features a new framework that creates semantically correlated memes and generates contextual embeddings for accurate meme segmentation.

Copied to clipboard

Weizhe Lin, Jingbiao Mei, Jinghong Chen, Bill Byrne

Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models (2025.acl-long)

Challenge:	Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA).
Approach:	They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks.
Outcome:	The proposed framework is used to train and evaluate multi-modal retrievers.

Copied to clipboard

Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Helen Li, Ziwei Liu, Kiyoharu Aizawa

Robustness in NLP

Challenge:	Multiple-choice question answering (MCQA) is widely used to assess the understanding capability of Large Multimodal Models (LMMs).
Approach:	They propose a task to evaluate the robust understanding capability of Large Multimodal Models (LMMs) they introduce a benchmark to assess performance across various ability dimensions .
Outcome:	The proposed model can withhold answers when encountering unsolvable problems of MCQA, proving it understands the answer.

Beyond Single Frames: Can LMMs Comprehend Implicit Narratives in Comic Strip? (2025.findings-emnlp)

Copied to clipboard

Xiaochen Wang, Heming Xia, Jialin Song, Longyu Guan, Qingxiu Dong, Rui Li, Yixin Yang, Yifan Pu, Weiyao Luo, Yiru Wang, Xiangdi Meng, Wenjie Li, Zhifang Sui

AdaptMerge: Inference Time Adaptive Visual and Language-Guided Token Merging for Efficient Large Multimodal Models (2025.findings-emnlp)

Challenge:	Large Multimodal Models have demonstrated strong performance on vision-language benchmarks, yet current evaluations focus on single-image reasoning.
Approach:	STRIPCIPHER is a benchmark designed to evaluate model ability on understanding implicit narratives in silent comics.
Outcome:	STRIPCIPHER is a high-quality, human-annotated dataset featuring fine-grained annotations and comprehensive coverage of varying difficulty levels.

Copied to clipboard

Zahidul Islam, Mrigank Rochan

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment (2024.acl-long)

Challenge:	Existing token reduction methods ignore image complexity and vision-language interactions, ignoring image complexity.
Approach:	They propose a training-free, inference-time token merging strategy that adaptively reduces visual tokens by leveraging feature diversity and language-guided relevance.
Outcome:	The proposed approach outperforms state-of-the-art token reduction methods on Google’s Gemma 3 models while achieving reduced computational costs and improved performance.

Copied to clipboard

Yunxin Li, Xinyu Chen, Baotian Hu, Haoyuan Shi, Min Zhang

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization (2024.findings-emnlp)

Challenge:	Recent Large Multimodal Models (LMMs) focus on visual knowledge-dimension alignment, but ignore visual knowledge.
Approach:	They propose a cognitive visual-language mapper that integrates visual-linguistic knowledge alignment with a fine-grained knowledge Adapter.
Outcome:	The proposed model significantly improves LMMs on knowledge-based visual question answering (VQA) it also improves the performance of other models, including GPT-4V and Gemini-Pro.

Copied to clipboard

Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

Audio Is the Achilles’ Heel: Red Teaming Audio Large Multimodal Models (2025.naacl-long)

Challenge:	Low-Rank Adaptation (LoRA) improves training efficiency by updating only a small portion of the weights in Large Language Models.
Approach:	They propose a rotation-aware scheme to fine-tune rotated outlier-free LLMs for effective weight-activation quantization.
Outcome:	The proposed method improves low-bit LoRA convergence and post-training quantization robustness.

Copied to clipboard

Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari

Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs (2025.naacl-long)

Challenge:	Large Language Models (LMMs) have demonstrated ability to interact with humans through text . however, safety of audio LMMs remains under-explored .
Approach:	They red team the safety of five audio LMMs under three settings . they find that audio Lmms suffer an average attack success rate of 69.14% on harmful questions .
Outcome:	a new study shows that audio LMMs suffer an average success rate on harmful questions . the authors also show that the models exhibit safety vulnerabilities when distracted .

Copied to clipboard

Anirudh Phukan, Divyansh Divyansh, Harshit Kumar Morj, Vaishnavi Vaishnavi, Apoorv Saxena, Koustava Goswami

Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection (2025.coling-main)

Challenge:	Large Multimodal Models are plagued by hallucinations that limit their reliability and adoption.
Approach:	They propose a method that leverages contextual token embeddings from LMMs to detect hallucinations.
Outcome:	The proposed method improves hallucination detection and grounding across diverse categories while excelling in tasks requiring contextual understanding.

Copied to clipboard

Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, Jiebo Luo

CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models (2024.acl-long)

Challenge:	Existing methods for detecting hateful memes rely on extensive training.
Approach:	They propose a method that integrates evolution attribute and in-context information of memes into large multimodal models via Chain-of-Evolution (CoE) prompting.
Outcome:	The proposed method improves existing methods on public datasets and can be used as interpretive tool to promote understanding of evolution of memes.

Copied to clipboard

Zixin Chen, Hongzhan Lin, Ziyang Luo, Mingfei Cheng, Jing Ma, Guang Chen

Sentiment Analysis

Challenge:	Current methods for multimodal sarcasm target identification focus on superficial indicators in an end-to-end manner, overlooking the nuanced understanding of multimodal content.
Approach:	They propose a multimodal sarcasm target identification framework with a coarse-to-fine paradigm by augmenting sarcasm explainability with reasoning and pre-training knowledge.
Outcome:	The proposed framework outperforms state-of-the-art methods and exhibits explainability in deciphering sarcasm as well.

Learning to Describe Implicit Changes: Noise-robust Pre-training for Image Difference Captioning (2025.findings-emnlp)

Copied to clipboard

Zixin Guo, Jiayang Sun, Tzu-Jui Julius Wang, Abduljalil Radman, Selen Pehlivan, Min Cao, Jorma Laaksonen

Navigating Hallucinations for Reasoning of Unintentional Activities (2024.findings-emnlp)

Challenge:	Large Multimodal Models (LMMs) are used to capture subtle differences between images but are noisy and coarse summaries.
Approach:	They propose a noise-robust approach to image difference capture using large multimodal models . they use LMMs with structured prompts to generate fine-grained change descriptions .
Outcome:	The proposed model outperforms streamlined architectures and improves inference efficiency.

Copied to clipboard

Shresth Grover, Vibhav Vineet, Yogesh Rawat

LEAF: Towards Lightweight Explainable Hateful Video Detection via Self-Grounding CoT Guided Stage-Wise Distillation (2026.findings-acl)

Challenge:	Existing models of intentionality recognition struggle to understand the reasoning behind unintentional actions.
Approach:	They propose a novel prompting technique which allows the model to navigate through hallucinated thoughts to achieve better reasoning.
Outcome:	The proposed prompting technique outperforms standard prompting while minimizing hallucinations.

Copied to clipboard

Jian Lang, Rongpei Hong, Meihui Zhong, Kaiju Li, Ting Zhong, Qiang Gao, Fan Zhou

ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)

Challenge:	Existing methods for detecting hateful videos rely on opaque models with no insight into their decisions.
Approach:	They propose a lightweight, explainable video detection framework that distills "explainability" from LMMs into efficient Smaller Multimodal Models (SMMs) they use a self-grounded chain-of-thought mechanism to generate unbiased supervision signals for videos .
Outcome:	The proposed framework outperforms existing methods in detection accuracy and explainability on three video benchmarks.

Copied to clipboard

Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, Cheng-Lin Liu

Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights (2024.emnlp-main)

Challenge:	Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency.
Approach:	They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task.
Outcome:	The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation.

Copied to clipboard

Hao Yang, Lizhen Qu, Ehsan Shareghi, Reza Haf

Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024.acl-long)

Challenge:	Large Multimodal Models have demonstrated a strong capability to understand multimodal information and interact with human users.
Approach:	They propose a speech-specific risk taxonomy to assess LMMs' ability to detect high-risk interactions in multimodal settings.
Outcome:	The proposed model is based on a speech-specific risk taxonomy covering 8 risk categories . it shows that the models are ineffective in detecting paralinguistic-specific risks in speech .

Copied to clipboard

Zihao Yue, Liang Zhang, Qin Jin

MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Models (2025.findings-acl)

Challenge:	Extensive research has shed light on the origins of multimodal hallucinations, including the inability of vision encoders to represent finegrained visual details and model reliance on inherent parametric knowledge such as language priors and statistical biases.
Approach:	They propose to use EOS to terminate generation of large multimodal models by comparing the generated text with the image to mitigate multimodal hallucinations.
Outcome:	The proposed method significantly improves the hallucination performance of Large Multimodal Models without additional data or knowledge.

Copied to clipboard

Bohan Jin, Shuhan Qi, Kehai Chen, Xinyi Guo, Xuan Wang

MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models (2026.findings-acl)

Challenge:	Large Multimodal Models (LMMs) have raised concerns about model toxicity.
Approach:	They propose a model to measure the toxicity gap between models and their hard level to determine whether they can handle dual-implicit toxicity.
Outcome:	The proposed model can handle dual-implicit toxicity effectively on 13 prominent LMMs, but its performance drops significantly in hard level.

Copied to clipboard

Kailin Jiang, Ning Jiang, Yuntao Du, Yuchen Ren, Yuchen Li, Yifan Gao, Jinhe Bi, Yunpu Ma, Bin Li, Lei Liu, Qing Li

Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency (2025.acl-long)

Challenge:	Existing benchmarks for Large Multimodal Models (LMMs) are constrained by static representations, inadequately evaluating their ability to understand time-sensitive knowledge.
Approach:	They propose a benchmark containing 2,104 time-sensitive knowledge samples spanning six knowledge types to evaluate temporal awareness along 6 key dimensions and 11 challenging tasks.
Outcome:	The proposed benchmark measures temporal awareness along 6 key dimensions and 11 tasks, while most open-source LMMs still lack time understanding ability.

Copied to clipboard

Jiafeng Liang, Shixin Jiang, Xuan Dong, Ning Wang, Zheng Chu, Hui Su, Jinlan Fu, Ming Liu, See-Kiong Ng, Bing Qin

Robustness in NLP

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance on general video comprehension benchmarks, but their robustness needs to be thoroughly investigated for broader applications.
Approach:	They propose a temporal robustness benchmark which introduces temporal inconsistency perturbations separately at the visual and textual modalities to assess the robustness of models.
Outcome:	The proposed method improves the model’s robustness and reliability in temporal analysis.

StatsChartMWP: A Dataset for Evaluating Multimodal Mathematical Reasoning Abilities on Math Word Problems with Statistical Charts (2025.findings-emnlp)

Copied to clipboard

Dan Zhu, Tianqiao Liu, Zitao Liu

MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique (2025.findings-emnlp)

Challenge:	StatsChartMWP is a dataset for evaluating visual mathematical reasoning abilities on math word problems with statistical charts.
Approach:	They propose a dataset for evaluating visual mathematical reasoning abilities on math word problems with statistical charts.
Outcome:	The proposed model is more effective than open-source approaches.

Copied to clipboard

Gailun Zeng, Ziyang Luo, Hongzhan Lin, Yuchen Tian, Kaixin Li, Ziyang Gong, Jianxiong Guo, Jing Ma

Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics (2025.findings-emnlp)

Challenge:	e MM-CRITIC is a holistic benchmark for evaluating the critique ability of Large Multimodal Models (LMMs) covering 8 main task types and over 500 tasks, covering 4471 samples.
Approach:	They introduce a holistic benchmark for evaluating the critique ability of Large Multimodal Models across multiple dimensions: basic, correction, and comparison.
Outcome:	The proposed benchmark covers 8 main task types and over 500 tasks and is composed of 4471 samples.

Copied to clipboard

Yuriel Ryan, Rui Yang Tan, Kenny Tsu Wei Choo, Roy Ka-Wei Lee

ELTLM: Evaluation of Longitudinal Temporal Large Multimodal Models in Clinical Scenarios (2026.findings-acl)

Challenge:	PixelHumor is a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences.
Approach:	PixelHumor is a benchmark dataset of 2,800 annotated multi-panel comics designed to evaluate LMMs’ ability to interpret multimodal humor and recognize narrative sequences.
Outcome:	Experiments with state-of-the-art LMMs reveal that top models achieve only 61% accuracy in panel sequencing, far below human performance.

Copied to clipboard

Gengyuan Hu, Haoxiang Liu, Chenhong Cao, Shilei Tan, Wei Gong

Aligning Large Multimodal Models with Factually Augmented RLHF (2024.findings-acl)

Challenge:	Existing evaluation benchmarks focus on static evaluation of large multimodal models . existing evaluation paradigms neglect a critical aspect of clinical practice: longitudinal analysis .
Approach:	They propose a temporal perception and reasoning benchmark to assess models' temporal grounding and consistency.
Outcome:	ELTLM features a hierarchical task taxonomy comprising Temporal Perception QA and Temporal Reasoning QA.

Copied to clipboard

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding (2024.emnlp-main)

Challenge:	Large Multimodal Models (LMMs) are built across modalities and the misalignment between two modality can result in "hallucination" . developing LMMs faces challenges such as a lack of data and a limited number of data sets.
Approach:	They propose a new algorithm that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options.
Outcome:	The proposed approach improves on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 and an improvement of 60% on MMHAL-BENCH over other baselines.

Copied to clipboard

Manan Suri, Puneet Mathur, Franck Dernoncourt, Rajiv Jain, Vlad Morariu, Ramit Sawhney, Preslav Nakov, Dinesh Manocha

MemeQA: Holistic Evaluation for Meme Understanding (2025.acl-long)

Challenge:	Document structure editing involves manipulating localized textual, visual, and layout components in document images based on user’s requests.
Approach:	They propose a framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs) by localizing edit regions of interest and disambiguating user edit requests into edit commands.
Outcome:	The proposed framework outperforms baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12%) tasks.

Copied to clipboard

Khoi P. N. Nguyen, Terrence Li, Derek Lou Zhou, Gabriel Xiong, Pranav Balu, Nandhan Alahari, Alan Huang, Tanush Chauhan, Harshavardhan Bala, Emre Guzelordu, Affan Kashfi, Aaron Xu, Suyesh Shrestha, Megan Vu, Jerry Wang, Vincent Ng

Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA (2025.findings-acl)

Challenge:	Existing benchmarks for meme understanding only concern narrow aspects of meme semantics.
Approach:	They propose to use multiple-choice questions to evaluate meme comprehension . they use a dataset of over 9,000 multiple-question questions to assess meme comprehension.
Outcome:	The proposed model outperforms existing models on meme comprehension . the model makes many errors on memes where proper understanding requires going beyond sentiment .

Copied to clipboard

Qianqi Yan, Xuehai He, Xiang Yue, Xin Eric Wang

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? (2025.acl-long)

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance on existing medical visual question answering benchmarks.
Approach:	They evaluate large multimodal models that perform worse than random guessing on medical questions . authors suggest more robust evaluation methods to ensure reliability of LMMs .
Outcome:	a new study shows that large multimodal models perform worse than random guessing on medical visual question answering benchmarks.

Copied to clipboard

Runqi Qiao, Qiuna Tan, Guanting Dong, MinhuiWu MinhuiWu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma GongQue, Shanglin Lei, YiFan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, Honggang Zhang

Error-driven Data-efficient Large Multimodal Model Tuning (2025.acl-long)

Challenge:	Existing benchmarks focus more on end-to-end performance, but neglect the underlying principles of knowledge acquisition and generalization.
Approach:	They propose a benchmark specifically designed to explore the problem-solving principles by decomposing 6.5K visual math problems into 10.9K step-level questions for evaluation.
Outcome:	The proposed benchmark covers 6.5K visual math problems and 10.9K step-level questions spanning 5 layers of knowledge granularity and 67 hierarchical knowledge concepts.

Copied to clipboard

Barry Menglong Yao, Qifan Wang, Lifu Huang

ExtremeAIGC: Benchmarking LMM Vulnerability to AI-Generated Extremist Content (2025.findings-emnlp)

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks, but task-specific tuning samples are often not readily available or expensive and time-consuming to obtain.
Approach:	They propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without extensive task-specific training samples.
Outcome:	The proposed model achieves an average performance boost of 7.01% on seven tasks across three training data scales and three different training datascales.

Copied to clipboard

Bhavik Chandna, Mariam Aboujenane, Usman Naseem

Robustness in NLP

Challenge:	Existing datasets for evaluating LMM robustness lack exploration of extremist content . existing models lack diverse image generation models and comprehensive coverage of historical events .
Approach:	They propose a benchmark dataset to assess LMM models against extremist content . ExtremeAIGC simulates real-world events and malicious use cases .
Outcome:	a new benchmark dataset and evaluation framework assesses LMM models against extremist content.

An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models (2024.emnlp-main)

Copied to clipboard

Fatemeh Shiri, Xiao-Yu Guo, Mona Far, Xin Yu, Reza Haf, Yuan-Fang Li

Reasoning

Challenge:	Large Multimodal Models (LMMs) have shown impressive generalization ability on vision and language tasks, but their spatial understanding is under-explored.
Approach:	They construct a VQA dataset to analyze LMMs' spatial reasoning capabilities.
Outcome:	The proposed model is stronger at basic object detection than complex spatial reasoning.

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)

Copied to clipboard

Jingbiao Mei, Jinghong Chen, Guangyu Yang, Weizhe Lin, Bill Byrne

Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling (2026.acl-long)

Challenge:	Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities.
Approach:	They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs.
Outcome:	The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs.

Copied to clipboard

Yujie Chen, Tailai Chen, Yifeng Gao, Zoe Wanying He, Yijue Xu, Shaobo Wang, Linfeng Zhang

Green & Sustainable NLP

Challenge:	Existing methods to reduce sequence length rely on heuristics that break compatibility with hardware-efficient kernels like FlashAttention.
Approach:	They propose a method that selectively halts stabilized tokens by monitoring layer-wise update dynamics of the self-attention mechanism.
Outcome:	The proposed method can reduce prefill complexity while preserving model accuracy and hardware efficiency.

Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos (2026.acl-long)

Copied to clipboard

Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xiang Yue, Bo Li, Yuanhan Zhang, Ziwei Liu

Visual Data in NLP

Challenge:	Existing video benchmarks do not evaluate the knowledge acquisition capabilities of Large Multimodal Models (LMMs) existing video benchmark focuses on static, general visual understanding tasks, without evaluating whether models can acquire knowledge dynamically.
Approach:	They propose a multi-modal, multi-discipline, multitrack benchmark that evaluates Large Multimodal Models’ ability to acquire knowledge from college-level, educational videos.
Outcome:	The proposed benchmark reveals a substantial gap between human learners and current Large Multimodal Models (LMMs) and focuses on improving their learning efficiency.

Promptception: How Sensitive Are Large Multimodal Models to Prompts? (2025.findings-emnlp)

Copied to clipboard

Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan

Tree-of-Evidence: Efficient "System 2" Search for Faithful Multimodal Grounding (2026.findings-acl)

Challenge:	despite the success of Large Multimodal Models, prompt design for MCQA remains poorly understood.
Approach:	They propose a framework for evaluating prompt sensitivity in LMMs . they propose 61 prompt types, each targeting specific aspects of prompt formulation .
Outcome:	The proposed framework evaluates 10 LMMs across 3 MCQA benchmarks.

Copied to clipboard

Micky C. Nnamdi, Benoit Louis Marteau, Yishan Zhong, J. Ben Tamo, May Dongmei Wang

Hidden in Plain Sight: Evaluation of the Deception Detection Capabilities of LLMs in Multimodal Settings (2025.acl-long)

Challenge:	Attention-based methods fail to faithfully represent the model’s decision process when integrating heterogeneous modalities.
Approach:	They propose an inference-time search algorithm that frames interpretability as a discrete optimization problem.
Outcome:	The proposed algorithm retains over 98% of full-model AUROC with as few as five evidence units and achieves higher decision agreement and lower error than LIME, SHAP, saliency, and concept-bottleneck baselines under sparse budgets.

Copied to clipboard

Md Messal Monem Miah, Adrita Anika, Xi Shi, Ruihong Huang

TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models (2026.findings-acl)

Challenge:	Detecting deception in an increasingly digital world is a critical and challenging task.
Approach:	They evaluate the performance of both open-source and proprietary LLMs on three datasets . they find that fine-tuned LLM achieve state-of-the-art performance on textual deception detection .
Outcome:	The proposed models achieve state-of-the-art on textual deception detection, whereas LMMs struggle to fully leverage multimodal cues.

Copied to clipboard

Hao Zhang, Lyu Mengsi, Chenrui He, Yulong Ao, Yonghua Lin

Glance-or-Gaze: Incentivizing LMMs to Adaptively Focus Search via Reinforcement Learning (2026.findings-acl)

Challenge:	Existing token pruning methods rely on costly calibration or suboptimal importance metrics, leading to redundant retained tokens.
Approach:	They propose a token pruning strategy that preserves cross-modal alignment and informational diversity.
Outcome:	The proposed method maintains strong performance while reducing tokens by 88.9% on two models.

Copied to clipboard

Hongbo Bai, Yujin Zhou, Yile Wu, Chi-Min Chan, Pengcheng Wen, Kunhao Pan, Sirui Han, Yike Guo

Activation Reward Models for Few-Shot Model Alignment (2026.findings-acl)

Challenge:	Existing search-augmented approaches rely on indiscriminate whole-image retrieval and lack deep iterative reflection, limiting their effectiveness on complex visual queries.
Approach:	They propose a fully autonomous framework that shifts from passive perception to active visual planning and introduces a Selective Gaze mechanism that dynamically chooses whether to glance at global context or gaze into high-value regions.
Outcome:	Experiments across six benchmarks demonstrate state-of-the-art performance.

Copied to clipboard

Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig

Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation (2026.acl-long)

Challenge:	A common approach is to use reward models that enable reinforcement-learning post-training.
Approach:	They propose a method that steers LLM activations to align with few-shot preference data without finetuning.
Outcome:	The proposed method surpasses zero-shot, few-shot and voting-based benchmarks on reward hacking and noise signals.

Copied to clipboard

Jiawei Zhou, Chi Zhang, Xiang Feng, Qiming Zhang, Haibo Qiu, Lihuo He, Dengpan Ye, Xinbo Gao, Jing Zhang

From Words to Pixels: A Comprehensive Survey on Large Language Models in Visual Segmentation (2026.acl-long)

Challenge:	a benchmark is designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code.
Approach:	They propose a benchmark to evaluate the capability of Large Multimodal Models to convert digital graphics into executable code.
Outcome:	The proposed benchmark exposes the performance gap among leading LMMs . the benchmark features 1130 meticulously curated samples .

Copied to clipboard

Yizhou Wang, Mang Tik Chiu, Lingzhi Zhang, Xuan Shen, Sohrab Amirghodsi, Yun Fu